Papers with semantic textual similarity tasks
AugCSE: Contrastive Sentence Embedding with Diverse Augmentations (2022.aacl-main)
Copied to clipboard
| Challenge: | Similar work has shown that a single augmentation can be used to learn a robust generalpurpose representation with contrastive learning. |
| Approach: | They propose a unified framework to utilize diverse sets of data augmentations to achieve a better, general-purpose sentence embedding model. |
| Outcome: | The proposed framework achieves state-of-the-art results on downstream transfer tasks and performs competitively on semantic textual similarity tasks, using only unsupervised data. |
Fine-tuning CLIP Text Encoders with Two-step Paraphrasing (2024.findings-eacl)
Copied to clipboard
| Challenge: | Contrastive language-image pre-training models have demonstrated considerable success across various vision-language tasks, such as text-to-image retrieval. |
| Approach: | They propose a fine-tuning approach to enhance the representations of CLIP models for paraphrases by leveraging large language models. |
| Outcome: | The proposed model improves on baseline models across paraphrased retrieval, visual genome relation and attribution, and seven semantic textual similarity tasks. |
RobustSentEmbed: Robust Sentence Embeddings Using Adversarial Self-Supervised Contrastive Learning (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing pre-trained language models exhibit poor generalization and robustness in adversarial settings. |
| Approach: | They propose a self-supervised sentence embedding framework that improves generalization and robustness against adversarial attacks. |
| Outcome: | The proposed framework reduces the success rate of adversarial attacks by almost half . it also improves semantic text similarity tasks and various transfer tasks . |
DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings (2022.naacl-main)
Copied to clipboard
Yung-Sung Chuang, Rumen Dangovski, Hongyin Luo, Yang Zhang, Shiyu Chang, Marin Soljacic, Shang-Wen Li, Scott Yih, Yoon Kim, James Glass
| Challenge: | Recent work shows that finetuning pretrained models with contrastive learning makes it possible to learn good sentence embeddings without labeled data. |
| Approach: | They propose an unsupervised contrastive learning framework for learning sentence embeddings . they use a masked language model to mask out the edited sentence . |
| Outcome: | The proposed framework outperforms SimCSE on semantic textual similarity tasks by 2.3 absolute points. |
Learning Semantic Textual Similarity via Topic-informed Discrete Latent Variables (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent discrete latent variable models have received a surge of interest in both NLP and CV . they are comparable to the continuous counterparts in representation learning, but are more interpretable in their predictions. |
| Approach: | They develop a topic-informed discrete latent variable model for semantic textual similarity . they inject the quantized representation into a transformer-based language model . |
| Outcome: | The proposed model outperforms strong baselines in semantic textual similarity tasks. |
Continual Learning for Sentence Representations Using Conceptors (N19-1)
Copied to clipboard
| Challenge: | Existing sentence encoders for distributed representations of sentences are limited in their performance on fixed corpora. |
| Approach: | They propose a continual learning scenario for distributed representations of sentences . they initialize sentence encoders with corpus-independent features and update them sequentially . |
| Outcome: | The proposed sentence encoder can learn features from new corpora while maintaining its competence on previously encountered corporales. |
Debiased Contrastive Learning of Unsupervised Sentence Representations (2022.acl-long)
Copied to clipboard
| Challenge: | Recent studies have shown that contrastive learning improves pre-trained language models to derive high-quality sentence representations. |
| Approach: | They propose a framework to punish false negatives and generate noise-based negatives to guarantee the uniformity of the representation space. |
| Outcome: | The proposed framework improves pre-trained language models while pushing apart irrelevant negatives to guarantee the uniformity of the representation space. |
MCSE: Multimodal Contrastive Learning of Sentence Embeddings (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing approaches to learning semantically meaningful sentence embeddings are limited by the complexity of pre-trained models. |
| Approach: | They propose a sentence embedding learning approach that exploits both visual and textual information via a multimodal contrastive objective. |
| Outcome: | The proposed approach improves the state-of-the-art average Spearman’s correlation by 1.7% on a variety of semantic textual similarity tasks. |
WhitenedCSE: Whitening-based Contrastive Learning of Sentence Embeddings (2023.acl-long)
Copied to clipboard
| Challenge: | Extensive experiments on seven semantic textual similarity tasks show our method achieves consistent improvement over the contrastive learning baseline and sets new states of the art. |
| Approach: | They propose a whitening-based contrastive learning method for sentence embedding learning which combines contrastive and shuffled group whitening. |
| Outcome: | The proposed method achieves better alignment and uniformity on seven semantic textual similarity tasks. |
On the Sentence Embeddings from Pre-trained Language Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Pre-trained contextual representations like BERT have been widely used for NLP tasks. |
| Approach: | They propose to transform anisotropic sentence embedding distribution to smooth and isotropic Gaussian distribution by normalizing flows that are learned with an unsupervised objective. |
| Outcome: | The proposed method achieves significant performance gains over state-of-the-art embeddings on a variety of semantic textual similarity tasks. |
Composition-contrastive Learning for Sentence Embeddings (2023.acl-long)
Copied to clipboard
| Challenge: | Recent work shows potential to learn vector representations from unlabelled data without task-specific fine-tuning. |
| Approach: | They propose to maximize alignment between textual embeddings and a composition of their phrasal constituents. |
| Outcome: | The proposed approach improves on similarity tasks comparable to state-of-the-art approaches. |
MATCHA: Matching Text via Contrastive Semantic Alignment (2026.findings-acl)
Copied to clipboard
| Challenge: | MATCHA is an automatic metric that rewards semantic agreement with a reference and penalizes contradictions. |
| Approach: | They introduce a metric that jointly rewards semantic agreement with a reference and penalizes contradictions. |
| Outcome: | The proposed metric outperforms popular metrics on eight public benchmarks compared with human annotations on question-answering, image caption generation, natural language inference, summarization, and semantic textual similarity tasks. |
SemPA: Improving Sentence Embeddings of Large Language Models through Semantic Preference Alignment (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing sentence embedding methods rely on fixed prompt templates or involve modifications to the model architecture, compromising its generative capabilities. |
| Approach: | They propose a sentence-level direct preference optimization approach that boosts the sentence representations while preserving the generative ability of LLMs. |
| Outcome: | The proposed method improves representations of semantically meaningful vectors without sacrificing generation capability. |